Building a portable gesture-to-audio/visual speech system
نویسندگان
چکیده
We have constructed an easy-to-use portable, wearable gesture-to-speech system based on the Glove-TalkII and GRASSP gesture-controlled speech systems and a vizeme based face-synthesizer. Our new portable system is called a Digital Ventriloquized Actor (DIVA) and refines the use of the formant speech synthesizer. Using a DIVA, a user can speak using hand gestures mapped to both synthetic sound and face using a mapping function that preserves gesture trajectories. By making DIVAs portable and self-contained, speakers can communicate with others in the community and perform in new music/theatre stage productions. DIVA performers also allow us to study the relationship between visible gestures and speech/song production.
منابع مشابه
Text to Avatar in Multi-modal Human Computer Interface
In this paper, we present a new text-driven avatar system, which consists of three major components, a text-to-speech (TTS) unit, a speech driven facial animation (SDFA) unit and a text-to-sign language (TTSL) unit. A new visual prosody time control model and an integrated learning framework are proposed to realize synchronization among speech synthesis, face animation and gesture animation, wh...
متن کاملThe processing of speech, gesture, and action during language comprehension.
Hand gestures and speech form a single integrated system of meaning during language comprehension, but is gesture processed with speech in a unique fashion? We had subjects watch multimodal videos that presented auditory (words) and visual (gestures and actions on objects) information. Half of the subjects related the audio information to a written prime presented before the video, and the othe...
متن کاملSeeing to hear better: evidence for early audio-visual interactions in speech identification.
Lip reading is the ability to partially understand speech by looking at the speaker's lips. It improves the intelligibility of speech in noise when audio-visual perception is compared with audio-only perception. A recent set of experiments showed that seeing the speaker's lips also enhances sensitivity to acoustic information, decreasing the auditory detection threshold of speech embedded in no...
متن کاملSpeech and manual gesture coordination in a pointing task
This study explores the coordination between manual pointing gestures and gestures of the vocal tract. Using a novel methodology that allows for concurrent collection of audio, kinematic body and speech articulator trajectories, we ask 1) which particular gesture (vowel gesture, consonant gesture, or tone gesture) the pointing gesture is coordinated with, and 2) with which landmarks the two ges...
متن کاملBrain regions differentially involved with multisensory and visual only speech gesture information
In this study a vowel identification task, controlling for intelligibility confounds, using audio visual stimuli at different signal to noise levels as well as visual only stimuli, is conducted to investigate neural processes involved with visual gesture information for speech perception. The fMRI results suggest that visual speech gesture information may serve to facilitate speech perception u...
متن کامل